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SUMMARY 


This descriptive study identifies the thinking and learning skills — such as good decision-making 
strategies and monitoring one’s own learning progress — that students should acquire, as described 
in standards documents from state departments of education, from national subject-area 
organizations, and from organizations concerned about adequate student preparation for post- 
secondary work. Because such skills are frequently embedded within standards that address 
subject-area content, it is often not clear which thinking and learning skills are commonly expected 
of students. This project seeks to clarify the scope and characteristics of these skills and to identify 
how they are addressed in subject-area standards from national organizations, as well as standard 
documents for the Central Region states. As states conduct their periodic review of standards for 
students, they seek better understanding of the skills and knowledge expected of students in other 
influential standards documents. In an effort to address this need, REL Central set out to 
examine these documents to determine what types of thinking and learning skills are commonly 
anticipated. 

Standards identify what students should know and be able to do by the time they graduate from 
high school, and what they need to accomplish along the way in order to meet these goals. 
Standards documents are the primary means by which expectations for students are communicated 
to educators, to students, and to parents, and have considerable influence on the content of the 
curriculum and day-to-day schooling. In order to understand what is expected of students at a 
national as well as a state level, then, standards documents provide a valuable source of 
information. 

This study asks the question, “What is the scope of the thinking 
and learning skills expected of students as identified in three 
significant sources of standards— state standards in the Central 
Region, subject-area standards from national organizations, and 
standards from post-secondary education across four major areas of 
study: English language arts, mathematics, science, and the social 
studies?” To conduct the review, analysts collected standards 
documents from the seven states in REL Central’s service area 
(Colorado, Kansas, Missouri, Nebraska, North Dakota, South 
Dakota, and Wyoming); from the national groups that have 
established specific standards in each of the four subject areas; and 

from organizations that have identified what graduating seniors should know in order to 
successfully pursue post-secondary schooling or advanced training. So that we could classify the 
content we found, and organize our findings - that is, communicate what types of thinking and 
learning skills appear in these standards documents - we selected and adapted an educational 
taxonomy (Marzano & Kendall, 2007) that provides useful definitions of and distinctions among 
these skills. Note that the taxonomy is simply a tool for analysis, and does not prescribe what 
students should learn; its value lies in the comprehensive and systematic way it defines a wide 
variety of skills related to thinking and learning. The taxonomy is organized in six levels - four in 
the area of cognition, one in metacognition, and one in the self-system. Each level is further 


An educational taxonomy is 
a means for organizing 
educational objectives for 
students. A taxonomy 
serves to identify key 
distinctions among 
different types of thinking 
and learning skills. 
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organized by type of mental process. So that, for example, the level of cognition called Knowledge 
Utilization includes the mental process of decision making. For the purposes of this study, another 
level, the topic level, was developed, in order to more specifically describe each process as it 
appears in the standards documents. For example, relative to decision-making, a standards 
document might focus specifically on the requirement that students carefully weigh alternative 
solutions to a problem. The weighing of alternative solutions was thus identified as one of the 
elements of decision-making that was addressed within the standards documents. Analysts, who 
have teaching backgrounds in the subject-area they reviewed and training in the taxonomy and 
methods of content-analysis, examined the standards documents for statements that expressed or 
implied thinking or learning skills that students should acquire. Upon finding any such content, 
they identified the taxonomic level and the process within that level that best described the skill. 
They completed the process by identifying the topic or chief element of that process that was 
addressed. Using such a method, analysts reviewed all the documents and assigned it to categories 
available in the taxonomy. The findings are discussed by cognition, or thinking skills; 
metacognition, or knowing about learning; and the self-system, or knowing about attention and 
motivation to learn. 

There are limitations to this study; chief among them is the nature of the analysis required to 
conduct it. References to thinking and learning skills were frequently not overt in the standards, 
but embedded within descriptions of student activities or demonstrations. It was often only 
through careful review of the standards that analysts identified the thinking and learning skills 
expected of students. More than one analyst reviewed each finding to confirm the analysis, but it is 
the case that documents were often not specific or deliberate about their identification of these 
skills. It is possible that there were thinking skills in these documents that were inadvertently 
omitted. It is also possible that, in the process of ensuring that no skills were missed, statements 
were interpreted as indicating that students should be engaged in a thinking skill, when there was 
no such intent on the part of the document’s authors. 

Cognition— Which Thinking Skills Are Expected of Students 

In our analysis of standards documents, we found that some levels of cognition are required 
universally, regardless of subject area and type of document: whether from states, subject-area 
organizations, or those that focus on preparation for post-secondary work. These cognitive skills 
include: retrieval, which includes recognizing and recalling facts and executing simple procedures; 
comprehension, which includes integrating information or symbolizing relationships; and analysis, 
which includes identifying similarities, classifying, forming generalizations, making predictions, 
and identifying errors. 

Our analysis of another type of cognition— knowledge utilization— revealed notable variation in 
coverage among the subject-areas. The cognitive process of knowledge utilization includes decision 
making, problem solving, experimenting, and investigating. It is beyond the scope of this study, 
which is entirely descriptive in purpose, to explain why such differences in cognitive processes 
might occur between subject areas or types of standards documents. Although readers may well 
draw their own conclusions (for example, that it is unsurprising that students in English classes are 
not expected to conduct experiments), the present task is limited to providing a sense of the scope 
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of processes in these documents and how they present themselves in the standards. Overall, our 
findings show that topics organized within knowledge utilization appear most frequently in science 
standards, and least frequently in social studies standards. 

Metacognition— What Students Are Expected to Know about How They Learn 

We found some evidence in standards documents that students are expected to engage in 
metacognition, which includes setting their own learning goals, monitoring their progress toward 
learning goals, and monitoring their thinking processes for accuracy and for clarity. As in the case 
with knowledge utilization, however, these expectations do not appear universally across all subject 
areas. 

Across all subject areas, students were most frequently expected to establish their own learning 
goals in the English language arts. The aspect of metacognition found to be most common across 
all subjects and documents concerns student monitoring of progress toward a learning goal; in 
other words, the student monitors how effectively he or she is attaining knowledge or skill. In the 
English language arts, for example, students are reminded to adjust strategies as necessary to 
ensure that they are meeting reading, writing, and listening goals. Nearly every document in 
mathematics, and most in science, also addressed this topic. 

Monitoring clarity— that is, the extent to which students monitor how clear they are about what 
they are learning— was found predominantly in the English language arts, in which clarity is 
emphasized as an important objective for reading comprehension; other subject-areas do not 
address reading comprehension. Monitoring accuracy, which refers to the attention students pay to 
their own accuracy and understanding while engaged in learning, was found principally in 
mathematics, where it appeared in all but one of the documents reviewed. Students are commonly 
reminded to check the reasonableness of their mathematical results and verify their work. 

Self- system— What Students Are Expected to Know about Their Attention 
and Motivation to Learn 

The standards documents indicated few expectations for students relative to the self-system, that is, 
the attitudes, beliefs, and emotions students hold about themselves as learners and about what 
they are learning. REL Central reviewed standards documents to determine whether students were 
expected to examine the importance of their learning, their efficacy as learners, their emotional 
response to learning, or their motivation to learn. 

Although we found few cases in which students are expected to consider the importance of what 
they are learning, the range of application for this aspect of the self-system was diverse. In the area 
of mathematics, for example, students are asked consider the importance of the mathematics they 
are learning to their own lives. They are also asked to consider the importance of knowing how 
they reached the solution to a particular problem. 

Within the standards documents reviewed, students are not typically expected to examine whether 
or not they are effective in their approach to learning. Such expectations commonly take the form 
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of asking students to consider whether they create barriers to their own learning, or if they allow 
external factors to derail their attention, and how they might remedy these distractions. 

Of all aspects included under the heading of the self-system, the expectation most frequently found 
in the documents was that students should be aware of their emotional response in the course of 
their learning. The common focus of this idea was that students should not allow emotions to 
cloud their judgment and should consider arguments dispassionately, allowing logic its proper 
role. 

In terms of motivation, there were no clear statements in the documents reviewed that students 
should examine their motivation to learn. However, a number of standards documents identified 
value in students demonstrating persistence, which analysts construed as being most closely related 
to motivation, of all categories identified within the self-system. Standards documents also made 
clear that students should have the time and opportunity to engage in work that is of interest to 
them, which suggests a concern that students have the means to connect to their learning in a way 
that motivates them. 

In summary, our review showed that there are expectations among the standards documents that 
students should demonstrate a variety of thinking and learning skills, and these may be organized 
under the categories of cognition, metacognition, and attention to the self as learner. Educators or 
state policymakers who wish to understand whether and how commonly students are expected to 
acquire cognitive, metacognitive, and self-system skills may find these results of interest.. 

Why this study? 

As states conduct their periodic review of standards for students, they seek better understanding of 
the skills and knowledge expected of students in other influential standards documents. In an 
effort to address this need, REL Central set out to examine a set of standards documents to 
determine what types of thinking and learning skills are commonly anticipated. In REL Central’s 
seven-state service area (Colorado, Kansas, Missouri, Nebraska, North Dakota, South Dakota, and 
Wyoming), educators and policy makers are committed to ensuring that students acquire the skills 
and knowledge essential for their future success. Our interviews of leadership groups of principals, 
superintendents, and chief state school officers all indicated that identifying essential learning for 
students is a priority. 

Standards serve the purpose of describing, with some specificity, what students should know and 
be able to do by the time they graduate from high school and in the grades along the way. 
Standards documents are the primary means by which these expectations are communicated to 
educators, to students, and to parents. State-level standards frequently have the force of legislation 
behind them and are readily available not only to teachers but to students, parents, and 
community members through postings on the Web sites of state education agencies. Standards 
developed by national subject-area organizations often help to inform the development of state 
standards. Recently, organizations that represent the interests of post-secondary organizations and 
the high-skills workplace also have promulgated their own standards to make clear what skills 
students should have in order to succeed in college or the workforce. 


Thinking and Learning Skills: What Da We Expect of Students? 


Page 4 



An earlier REL Central Issues & Answers report, based on a related concern for student 
preparation, examined whether or not the seven states in the region (Colorado, Kansas, Missouri, 
Nebraska, North Dakota, South Dakota, and Wyoming) expect all high school students to obtain 
the knowledge and skills identified as important for success in college or work. That study, “High 
School Standards and Expectations for the Workplace,” (Kendall, Pollack, Schwols, & Snyder, 
2006) focused on the extent to which state standards cover mathematics and the English language 
arts in the Central Region relative to expectations held by post-secondary institutions and the high- 
skills workplace, focusing on the academic content of the disciplines. 

No study has yet examined standards documents to determine what expectations are held for 
students beyond the academic content of the disciplines— specifically, what expectations are held 
for students related to thinking and learning skills. 

All seven Central Region states have published standards in each of the four core subject areas; 
these documents have varying but indisputable impact on the curriculum in the schools of each 
state. The standards documents from national subject-area groups, although many are more than a 
decade old, continue to be cited as influential in the development and revision of state standards 
and remain the sole national consensus among educators regarding important content for any 
subject area. Nearly all of these standards documents are readily available online for use by 
educators. For expectations held for graduating seniors by post-secondary institutions and the 
world of the high-skills workplace, three organizations have identified important English language 
arts and mathematics content; one organization addresses science and the social studies content as 
well. 

This report expands the question of academic expectations for all students to consider the 
processes related to learning— that is, cognitive and metacognitive processes, as well as processes 
related to the selfsystem— that may appear across core subject areas (English language arts, 
mathematics, science, and the social studies) from three sources. We reviewed standards from state 
departments of education and national subject-area groups, as well as reports on what students 
should know to be successful in post-secondary education and the high-skills workplace (see 
Appendix B, Table Bl). 

In order to identify the types of thinking and learning skills that appear in these standards 
documents, it is of course necessary to choose a method to organize the range of possible topics 
that are addressed. Researchers and scholars have published a number of papers and studies in 
attempts to define the nature and scope of critical thinking skills in addition to those skills that are 
required for students to take active steps to improve their learning and monitor their current levels 
of understanding (Bransford & Schwartz, 1999). Attention to one’s own learning, sometimes 
called process monitoring, is among a set of metacognitive and related skills that students must 
acquire in order to learn and adapt to the changing demands of school and the workplace. A 
student’s motivation, part of the self-system, is also among the set of skills that are critical to 
student’s learning (Wang, Haertel, & Walberg, 1992). There appears to be little consensus among 
the studies regarding what each skill set comprises. This is especially true regarding the 
metacognitive and self-system, for which definitions in the literature often overlap; in fact, either may 
appear described as a component of the other (Mosley, et al., 2005; further discussion on how 
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these ideas appear in research and theoretical studies appears in appendix A.) In order to identify 
the full range of skills that may be considered, Marzano and Kendall’s recent taxonomy (2005) was 
selected as a useful means for distinguishing among these systems and identifying the possible 
subcomponents of each (see the “Tool for Analysis” section for definitions of the major categories 
used in this report). 

Some assert that these “learning-to-learn skills,” may be inadequately addressed in academic 
standards and the curriculum (Cornford, 2002; White & Frederiksen, 2005). Whether or not this 
is true, it is the case that until we understand whether such expectations are present in current 
standards, it is not possible to consider whether they are sufficiently addressed or need to be 
included more systematically. 

The information gleaned by reviewing how well learning-to-learn skills are addressed in state-level 
standards, as well as in expectations held by post-secondary and workplace institutions, is provided 
for leaders responsible for establishing expectations at the school level (i.e., high school principals, 
superintendents, and local board chairs); state officials who set graduation standards (i.e., 
education agency personnel, board members, and legislators); and post-secondary faculty. In short, 
this report is intended to inform educators and policymakers who seek to understand what skills 
students are currently expected to acquire related to cognitive, metacognitive, and self-system skills, 
and how their own state compares to others, to standards established by national organizations, 
and to the expectations held by post-secondary institutions and the high-skills workplace. How, 
then, do standards documents address learning-to-learn skills? 

The Process of Analysis 

In order to conduct an analysis of types of thinking and learning skills, we required a system that 
could allow us to classify and organize any type of thinking and learning skills that might be found 
in the text of the standards, regardless of subject area or authoring organization. We reviewed 
possible educational taxonomies that might serve to identify key distinctions among different types 
of thinking and learning skills (see Appendix A for a discussion of this task) we selected and 
adapted an educational taxonomy (Marzano &. Kendall, 2007) that proved useful both for the 
analytic work and for organizing and presenting the findings of our work. Note that the taxonomy 
is simply a tool for analysis, and does not prescribe what students should learn; its value lies in the 
comprehensive and systematic way it defines a wide variety of skills related to thinking and 
learning. The taxonomy is organized in six levels - four in the area of cognition, one in 
metacognition, and one in the self-system. Each level is further organized by type of mental 
process. So that, for example, the level of cognition called Knowledge Utilization includes the 
mental process of decision making. For the purposes of this study, another level, the topic level, 
was developed, in order to more specifically describe each process as it appears in the standards 
documents. For example, relative to decision-making, a standards document might focus 
specifically on the requirement that students carefully weigh alternative solutions to a problem. 

The weighing of alternative solutions was thus identified as one of the elements of decision-making 
that was addressed within the standards documents. Analysts, who have teaching backgrounds in 
the subject-area they reviewed and training in the taxonomy and methods of content-analysis, 
examined the standards documents for statements that expressed or implied thinking or learning 
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skills that students should acquire. Upon finding any such content, they identified the taxonomic 
level and the process within that level that best described the skill. They completed the process by 
identifying the topic or chief element of that process that was addressed. Using such a method, 
analysts reviewed all the documents and assigned it to categories available in the taxonomy. The 
findings are discussed by cognition, or thinking skills; metacognition, or knowing about learning; 
and the self-system, or knowing about attention and motivation to learn. (For a description of the 
protocol, see Appendix C) 

The report will present these findings by each of the three systems in the taxonomy used to 
conduct the analysis — cognitive, metacognitive, and self-system. 

COGNITION - WHAT THINKING SKILLS ARE EXPECTED OF 
STUDENTS 

In order to identify the thinking and learning skills within the standards documents, we used a 
taxonomy that treats cognition as a system of four levels: retrieval, comprehension, analysis, and 
knowledge utilization. Each level comprises a set of processes. For example, the retrieval level 
includes the processes of recognizing, recalling, and executing (for a complete list of levels and 
processes, see Table 1, page 18). Analysts reviewed all the standards documents for the presence of 
such skills. As they did so, they noted the various ways in which students were expected to 
demonstrate these skills. This is an example on how the taxonomy was adapted for the purposes of 
this analysis. This level of organization does not exist in the taxonomy, but was added in order to 
create categories that more closely reflected the kinds of expectations that appeared in the 
standards. For example, the analysis level includes the process of analyzing errors. Analysts 
reviewing the standards identified a number of different ways in which students were asked to 
analyze errors, such as that they evaluate the accuracy of data, or review content for bias, or 
critique their own work for flaws. For each process analysts identified a set of topics or elements 
that seemed to recur across documents. This set of topics or elements provided a means for 
indicating how any one document touched on the aspects of a process that was found to be 
present across most of the documents. As described below, at the first three levels — retrieval, 
comprehension, and analysis — standards documents did not vary significantly in the ways the 
expected students to apply a mental process to demonstrate their learning. For the process of 
knowledge utilization, however, differences among the documents were more notable. For 
example, in decision-making, one document may emphasize only the selection of an appropriate 
tool or method for a given problem, while another may focus on how making a choice often 
requires an understanding of the risks and benefits involved. The following section describes all 
four levels within cognition. 

Level 1: Retrieval 

The first level of the taxonomy is the simplest aspect of learning: Retrieval. Retrieval refers to 
students’ recognition of information, or recall of it when asked. Retrieval also includes simple 
execution tasks: that is, tasks in which students recall and execute a series of steps, such as is the 
case in addition, that become routine or automatic. Standards, across document types and 
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subjects, do not vary in any notable way in their expectations that students have a mastery of basic 
facts and simple procedures. 

Level 2: Comprehension 

The second level of the taxonomy, Comprehension, concerns students’ ability to integrate 
knowledge or to represent their integration of knowledge symbolically. For the most part, emphasis 
on integration of knowledge appears to be common across documents, with no significant 
differences across document types. For the aspect of comprehension within the taxonomy that 
addresses symbolic or visual representation, however, there are some slight differences. State 
standards documents in the English language arts emphasize the use of visual organizers, but 
documents for post-secondary readiness do not. Across all document types, the greatest variety of 
ways in which students are asked to represent information appears within mathematics, largely 
owing to a focus on spatial reasoning and visualizing problems. Science follows next, with a focus 
on the use of models. For the social studies, attention to the symbolic aspect of comprehension 
relates to students’ understanding and use of maps and charts. 

Level 3: Analysis 

The third level of the taxonomy, Analysis, is comprised of matching, classifying, analyzing errors, 
generalizing and specifying (for a discussion of these and all other levels, consult appendix B.) No 
differences among subject areas or document types appear to be significant. All topics are 
addressed to some degree. 

Level 4: Knowledge Utilization 

Some differences among expectations for students appear when students are asked to apply their 
knowledge or understanding; in the taxonomy used for this study, this area is known as Knowledge 
Utilization. Knowledge utilization comprises four areas: decision making, problem solving, 
experimenting, and investigating. Figure 1 indicates this variation of coverage in the four subject 
areas for each of the authoring organizations in the area of knowledge utilization. Each process of 
knowledge utilization is discussed immediately below. 

Across all subjects and authoring organizations, decision making appears to be the aspect of 
knowledge utilization that is most commonly addressed. For example, a common sub-topic in this 
area is that students should be able to decide among available strategies— typically, from among 
strategies for reading in the English language arts, or for problem solving in mathematics and 
science. When decision making appears in the social studies, it is commonly centered on economic 
decision-making. 

In mathematics problem solving was addressed in the greatest variety of ways; this is true whether 
the standards are authored by a state department of education, the national subject-area group, or 
organizations focused on preparation for post-secondary work. Science also addresses a number of 
aspects in problem-solving. This topic also appears in the English language arts, though obliquely; 
for example, problem-solving is identified as one purpose that can be addressed through reading, 
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writing, speaking, and listening. The references to problem solving in the social studies center on 
the idea that complex issues and problems are best approached through understanding multiple 
perspectives. 

Experimenting, as defined by the taxonomy used for this analysis, involves generating and testing 
hypotheses about a specific physical or psychological phenomenon. A defining feature of 
experimenting tasks is that data are collected by the student. It is not surprising, then, that nearly 
all references to experimenting of this type across the documents appear in the subject area of 
science. Rare references in mathematics or social studies address the use of statistical analyses and 
surveys. 

Investigating is also best represented in science, but is also represented in the English language 
arts. Investigating, as defined in the taxonomy, does not depend on observable data but on 
assertions and opinions. In this view of investigating, logical argument plays an important role, 
and is the focus found among standards for the social studies. 
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How to Read the Figures 


Each column identifies a standards document examined for this report; each row 
identifies a mental process within the taxonomy. Analysts searched for evidence of each 
process within all the documents. There were often a number of aspects or elements in a 
process that could be found in one document, but it was often the case that not every 
element could found in every document. For example, some documents focused on the 
fact that students should identify a problem constraint, while in others, the constraint was 
given. In some documents, students were asked to identify assumptions before solving a 
problem; in other cases, they were not. Each of these aspects was counted as an element 
within the process. The intersection of row and column in the graphic indicates how 
extensively, compared to all other documents, the identified document addressed all the 
elements that was found associated with that process, here the process of problem solving. 
An empty circle indicates that a document did not address any of the elements associated 
with problem solving. A filled circle indicates that the document addressed the topic in as 
many different ways as appeared in all other documents. Topics varied in the number of 
ways they were addressed in the documents— this is indicated by the “number of elements” 
column next to each topic. For example, analysts found that nine different elements of 
problem solving could be identified across all documents and subject areas. The excerpt in 
Figure 1 represents the results of analyzing standards documents in the English language 
arts for their treatment of problem solving. In this analysis, three college work readiness 
documents were analyzed (in blue in the graph), one document from the national subject- 

area group (in maroon), and seven documents from the Central Region (yellow). 

National 

College/work subject-area 

tt 0 f readiness document(s) document(s) Central region state document(s) 

Elements Cl C2 C3 SI S2 S3 S4 S5 S6 

Problem 
solving 


Figure 1. Problem solving in the English language arts. 


Figure 1 indicates that two state documents (S4 & S5) did not reference problem-solving 
skills in standards for the language arts. State document S6 addressed one just one 
aspect— in this case, the document identified students’ use of multiple resources to address 
a research problem. A college readiness document (C3) similarly addressed one aspect of 
the topic. Hence, the wedge at the intersection of C3 and S6 and problem solving, 
graphically represents that each document addressed one of the nine elements. Across all 
documents, the number of aspects or elements for a given topic ranged from as little as 
one— in experimenting— to as many as nine elements, as in problem solving. The portion 
filled of each circle reflects how many of those elements a specific document addressed for 
the identified subject area. (See Table B1 for the codes used in each table to identify the 
documents examined for this report.) 
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Figure 2 represents how each given document, by authoring agency, within a subject area, 
addresses these four processes within the cognitive level called knowledge utilization. Some mental 
processes— such as experimenting— are somewhat narrowly defined and found to be either 
completely addressed within a document or completely absent; thus, the associated circle in the 
graph is either completely filled or empty. Other areas, such as decision making, had a variety of 
aspects or topics associated with them in the standards documents reviewed (for a discussion, see 
How the taxonomy of educational objectives was used to classify content related to cognition, metacognition 
and the self-system in the “Tool for Analysis” section.)- The degree to which a circle is filled reflects the 
variety of aspects found in the document for that subject when compared against all other 
documents for that subject. 
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Figure 2. Extent to which expectations for students relative to knowledge utilization appear in a 
varitey of standards documents in English language arts, mathematics, science, and social studies 

Note: Each column represents a document; the document type is indicated by the color key. For each process, the 
portion of the filled circle represents how much a document contains in comparison to the variety of ways in which 
the same process is addressed across all documents in that category. 

METACOGNITION - WHAT STUDENTS ARE EXPECTED TO KNOW 
ABOUT HOW THEY LEARN 

Metacognition can be described as knowledge or awareness of learning. It may refer to the process 
of monitoring one’s cognition or the knowledge one has about strategies for learning or 
characteristics of a task that affect cognition. For purposes of identifying the kinds of expectation 
held for students in the standards documents, metacognition is divided into four areas of student 
activity: specifying goals, monitoring process, monitoring accuracy, and monitoring clarity. Each 
aspect will be addressed in turn. In order to identify the thinking and learning skills within the 
standards documents, we used a taxonomy that divides metacognition into four areas of student 
activity: specifying goals, monitoring process, monitoring accuracy, and monitoring clarity. 
Documents were analyzed in order to identify how whether they addressed these areas. 

In order to capture the full range of possible topics identified from the research literature, analysts 
adapted the taxonomy in order to analyze metacognition for the extent to which it addressed 
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subject-specific and non-subject-specific metacognitive knowledge and processes. In other words, 
documents may (and did) address specifying goals as either generally useful, for example, or as 
particularly useful for the subject area at hand; similarly, standards could (and did) emphasize 
students’ engagement in the activity of monitoring their work for accuracy as they engaged in a 
mental activity, or, alternatively, identify that students should know or understand the value there 
is in monitoring their activity for accuracy. The findings are represented graphically in Figure 3. 

Specifying Goals 

The metacognitive process of specifying goals involves setting specific goals relative to one’s 
understanding or skill and developing a plan for accomplishing those goals. Very few documents, 
regardless of subject area or authoring agency, expect students to be able to set specific learning 
goals. In the English language arts, such references focus on personal improvement, such as 
setting a goal for reading or setting a goal for improved communication. The national subject area 
document for mathematics notes that “Students learn more and learn better when they can take 
control of their learning by defining their goals and monitoring their progress” (p. 21). 

Monitoring Process 

The area of metacognition for which standards documents most commonly hold expectations is 
students’ ability to monitor their own work or mental processes. English language arts standards 
showed the most variety in addressing this skill, focusing specifically on student self-reflection 
during the reading and writing process. These same standards also were the most likely to address 
process monitoring as an explicit part of the learning process. For example, the national subject 
area group states that the conscious process of learning how to learn is an essential element in 
students' language arts education. A number of expectations for students focus on their adjusting 
strategies as needed during the process of reading, writing, and listening. All such expectations 
concerned students’ contemporaneous monitoring of their learning process. However, what might 
be termed a post facto approach to process monitoring was also found within the documents. 
Documents in mathematics, in a few cases, value students’ reflection on their past thinking as a 
way to improve the problem-solving process. Science and social studies also had a few statements of 
this type. All such cases regarded the process itself as the topic for learning; that is, the expectation 
was that students should examine their past actions, rather than conduct “real-time” monitoring of 
their activities. For example, students were asked to consider and evaluate the processes they had 
used in recognizing and solving problems which, broadly understood, could be interpreted as an 
expectation that students revisit— monitor after the fact— the process they employed in the solution 
of a problem. This aspect represents fully a third of the items identified in this section. 

Monitoring Clarity 

Monitoring clarity involves the extent to which the student maintains clarity about a skill or 
knowledge. Most documents in the English language arts addressed this topic— not surprisingly, 
because monitoring the reading process is principally about ensuring that one has a clear 
understanding of what one is reading. In mathematics, the citations are general statements 
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enjoining teachers to ensure that students continue to refine and clarify their thinking. The 
national science documents, but no other documents, encourage students to ask specific clarifying 
questions as the work, such as “How certain are you of those results?”, a question that might be to 
encourage students to monitor for how clearly they understand the skill or knowledge. 

Monitoring Accuracy 

Monitoring accuracy, which addresses the extent to which the student determines he or she is 
accurate about specific knowledge, appears at least once across all subject areas. In the English 
language arts, the standards documents focus on the accuracy of student’s understanding of the 
text— that is, self-correcting for meaning, especially to ensure that bias has not skewed one’s 
understanding of what has been read. In mathematics, students are reminded to check the 
reasonableness of results during the execution of a process. In science, a shared idea among the 
documents is that the careful review of one’s completed process to verify results encourages careful 
checking of one’s thinking, which includes monitoring for bias. In the social studies students are 
encouraged to monitor accuracy by analyzing issues from multiple and historically objective 
perspectives. 
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Figure 3. Extent to which expectations for students relative to metacognition appear in a varitey 
of standards documents in English language arts, mathematics, science, and social studies 

Note: Each column represents a document; the document type is indicated by the color key. For each process, the 
portion of the filled circle represents how much a document contains in comparison to the variety of ways in which 
the same process is addressed across all documents in that category. 

SELF- SYSTEM - WHAT STUDENTS ARE EXPECTED TO KNOW 
ABOUT THEIR ATTENTION AND MOTIVATION TO LEARN 

The self-system can be described as the principal and first level of student engagement: that is, 
when or whether a student chooses to learn is a necessary first step for learning. Only if the self- 
system is engaged and the student has enlisted in an effort to learn is there sufficient reason to 
invoke the metacognitive system and establish or learning goals relative to knowledge or skill of 
interest. The taxonomy used for this analysis identifies the self-system as a set of interrelated 
attitudes, beliefs and emotions, the interaction of which determine an individual’s motivation and 
attention. There are four self-system processes: examining importance, examining efficacy, 
examining emotional response, and examining motivation. The findings are represented 
graphically in Figure 4. 
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Examining Importance 


The process of examining the relative importance of a task or undertaking is expected of students 
in just a few standards documents, although at occurs at least once in each subject area. It 
commonly appears in the form of students being asked to consider the importance of an activity; 
for example, the importance of asking questions, of reading a variety of texts, of reflecting on their 
thinking, and learning from their mistakes. Subject-specific concerns also appear in this category, 
such as students understanding of the importance of mathematics in their own life, and for the 
social studies, the value of understanding historical arguments in order to better understand 
present-day issues and concerns. 

Examining Efficiency 

Examining efficacy — that is, attending to the task of improving one’s knowledge or skill in a 
particular area — differs from process monitoring in that the central question is not observing to 
how knowledge or skill is used, but how effectively one manages the acquisition of knowledge and 
skill. This category, along with the category on examining importance just described, revealed the 
fewest expectations for students across all subject areas and authoring agencies. As in almost all 
categories, analysts often were required to make inferences in order to identify the kind of 
examination that appeared to be required. For example, under examining efficacy, knowing when 
to ask for help was taken as an indication that students are expected to examine how well they are 
managing a task. 

Examining Emotional Response 

Content related to examining emotional response also required that analysts make certain 
inferences from the text. For example, in the subject-area of science, more than one document 
notes that it is helpful for students to develop a tolerance for ambiguity. We inferred that such an 
activity requires that students resolve emotional responses in situations that might otherwise be 
frustrating. More direct descriptions related to examining the emotions are commonly found in 
language arts texts, for example, when students are asked to understand how their emotions shape 
their response to texts. A less direct example appears when students are asked how a particular 
speech made them feel, and thereby analyze how the speech appealed to their emotions. 

Examining Motivation 

In the category of examining motivation analysts often inferred an expectation in the documents 
based on closely related content. For example, an expectation that on students’ develop 
perseverance suggests that students should be well motivated. There were no explicit expectations 
that students examine their motivations directly, but motivation was valued for students 
(“persistence is vital”) and how students might become motivated was clearly a concern, as made 
clear in the observation that students should have the time and opportunity to investigate scientific 
questions that interest them. Some standards touched on the value of persisting in the face of 
difficulties; this appeared most often in the mathematics standards. 
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Figure 4. Extent to which expectations for students relative to self-system appear in a varitey of 
standards documents in English language arts, mathematics, science, and the social studies. 

Note: Each column represents a document; the document type is indicated by the color key. For each process, the 
portion of the filled circle represents how much a document contains in comparison to the variety of ways in which 
the same process is addressed across all documents in that category. 

IMPLICATIONS 

This study provides an overview of expectations held for students regarding thinking and learning 
skills as they are expressed in standards documents that currently in use or influential at the state 
and national levels. Standards documents commonly represent the consensus of many educators. 
This agreement among experts in the field accounts, in part, for the authority and influence of 
standards in the day-to-day curriculum. It should be noted, however, that standards documents 
that identify subject-area content typically communicate specific statements about student 
knowledge and skill that are unambiguous and readily understood. Identifying skills related to 
thinking and learning in these standards documents often required a close analysis of the text and 
careful inferences on the part of analysts; examples of such inferences have been provided through 
this report. The level of inference required to complete the analysis indicates that there remains 
some uncertainty about the actual extent of standards expectation for this area, and additional 
studies may be warranted to confirm or disconfirm what has been identified here. 

TOOL FOR ANALYSIS: A TAXONOMY OF EDUCATION OBJ ECT1VES 

In order to determine whether a taxonomy of educational objectives could be used to identify and 
code content in the standards documents, REL Central examined research reviews and theoretical 
studies related to cognition, metacognition, and the self-system to determine the potential scope of 
these topics. The review served not to reconcile views on these topics comprise, but to determine 
whether, given the range of topics found organized beneath these topics across a variety of 
theoretical studies and research surveys, a taxonomy could be successfully adapted to capture the 
full range of ways in which such content might appear in standards documents. In the course of 
this study, the suitability of other taxonomies to this purpose was also considered. (See Appendix 
A for a discussion.) A taxonomy of educational objectives from Marzano & Kendall (2007) was 
adapted to serve as the tool by which to identify to whether and to what degree students were 
expected to engage in a variety of activities at the cognitive, metacognitive, and self-system levels (see 
Figure 5 for an overview of the taxonomy and Table 1 for a description of the levels). 

The taxonomy of Marzano & Kendall used for this study is described in Table 1; an overview of 
the structure is provided in Figure 5. Among the advantages of the taxonomy for this study are the 
distinctions it provides among the cognitive system, the metacognitive system, and the self-system. 
As discussed in Appendix A, aspects of each of these systems appear in recent research literature 
and theoretical work on metacognition, motivation, and affect. Of interest, as well, is that the 
taxonomy is hierarchical by system, based on control and flow of processing. Briefly, the self-system 
can be described as the first level of student engagement: that is, when or whether a student 
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chooses to learn is a necessary first step for learning. Only if the self-system is engaged and the 
student has determined to engage in the effort to learn is it reasonable to expect that he or she will 
take a metacognitive interest in the effort - that is, to invest the energy required to monitor how 
well a task is being executed, whether that be a low-level skill or a higher-level thought process such 
as decision making. That is, under the direction of the metacognitive system, the elements of the 
cognitive system can then be monitored. The cognitive system includes processes ranging 
something as simple as retrieving information to the task of applying knowledge in an unfamiliar 
context. 

The three systems within the taxonomy are also hierarchical relative to the level of awareness 
required to effectively control their execution. Cognitive processes require a degree of awareness, 
of conscious thought, on the part of students -although this may be minimal in the case of routine 
addition. Observing the cognitive process -whether to check for accuracy, to question whether the 
task can be done more efficiently, and the like -requires more conscious effort, which is why the 
metacognitive system is considered more demanding. Finally, self-system processes such as 
examining importance of the task, checking one’s emotional response, and questioning one’s 
motivation for a task, represent a level of introspection and conscious thought that places still 
greater demands on a student’s attention. 



Levels of Processing 
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Figure 5. The organization of the taxonomy. Reprinted from Marzano, Robert & Kendall, John 
(2007). The New Taxonomy of Educational Objectives. 

Table 1. The cognitive, metacognitive, and self-systems. Adapted from Marzano, Robert & 
Kendall, John (2007). The New Taxonomy of Educational Objectives. 


Level 1: Retrieval (Cognitive System) 

Recognizing 

The student recognizes features of information but does not necessarily understand the structure of the knowledge 
or can differentiate critical from noncritical components. 

Recalling 

The student produces features of information but does not necessarily understand the structure of the knowledge 
or can differentiate critical from noncritical components. 

Executing 

The student performs a procedure without significant error but does not necessarily understand how and why the 
procedure works. 

Level 2: Comprehension (Cognitive System) 

Integrating 

The student identifies the basic structure of knowledge and the critical as opposed to noncritical characteristics. 

Symbolizing 

The student constructs an accurate symbolic representation of the knowledge differentiating critical and 
noncritical components. 

Level 3: Analysis (Cognitive System) 

Matching 

The student identifies important similarities and differences between knowledge. 

Classifying 

The student identifies superordinate and subordinate categories related to knowledge. 

Analyzing Errors 

The student identifies errors in the presentation or use of knowledge. 

Generalizing 

The student constructs new generalizations or principles based on knowledge. 

Specifying 

The student identifies specific applications or logical consequences of knowledge. 

Level 4: Knowledge Utilization (Cognitive System) 

Decision Making 

The student uses the knowledge to make decisions or makes decisions about the knowledge. 
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Problem Solving 

The student uses the knowledge to solve problems or solves problems about the knowledge. 

Experimenting 

The student uses the knowledge to generate and test hypotheses or generates and tests hypotheses about the 
knowledge. 

Investigating 

The student uses the knowledge to conduct investigations or conducts investigations about the knowledge. 

Level 5: Metacognitive System 

Specifying Goals 

The student establishes a goal relative to the knowledge and a plan for accomplishing the goal. 

Process Monitoring 

The student monitors the execution of a specific goal as it relates to the knowledge. 

Monitoring Clarity 

The student determines the extent to which he or she has clarity about the knowledge. 

Monitoring Accuracy 

The student determines the extent to which he or she is accurate about the knowledge. 

Level 6: Self-System 

Examining Importance 

The student identifies how important the knowledge is to him or her and the reasoning underlying this 
perception. 

Examining Efficacy 

The student identifies beliefs about his or her ability to improve competence or understanding relative to 
knowledge and the reasoning underlying this perception. 

Examining Emotional 
Response 

The student identifies emotional responses to knowledge and the reasons for these responses. 

Examining Motivation 

The student identifies his or her overall level of motivation to improve competence or understanding relative to 
knowledge and the reasons for this level of motivation. 


The taxonomy is organized by knowledge domain as well as the cognitive system, the metacognitive 
system, and the self-system (Figure 5). The knowledge domain comprises information, mental 
procedures, and psychomotor procedures. For the purpose of the analysis, it is not necessary to use 
categories of the knowledge domain, but it should be noted that the knowledge domain identifies 
the type of content on which the processes identified in the three systems operate. Thus, a learner 
may retrieve or comprehend (a process of the cognitive system) information about content that, in 
almost all models of thinking and learning, would be characterized as metacognitive (such as the 
knowledge that approaches to listening should vary by purpose, setting, and content). Normally, 
the taxonomy employed here would organize this content under the process of cognition 
(comprehension), not as metacognition, because it is comprehension process that is expected of 
the student, even though it is comprehension about a metacognitive process. However, because the 
taxonomy is being adapted in this study to organize and display all information related to 
metacognition and self-system, it has been altered to better suit these needs. In the example 
provided above, content that relates to metacognition, regardless of the type of mental process 
involved, is categorized under the metacognitive system. The fact that it may be actually more 
appropriately described as part of the knowledge domain is ignored. 

A similar case occurs in the self-system. Consider the observation that students “should 
understand math is an academic activity that requires time, sustained engagement, patience, and 
persistence” from the Standards for Success document (Conley, 2004, p. 31). This comment could 
not be categorized in the taxonomy as a self-system process, yet it addresses many aspects that the 
research literature associates with the self in learning. In the Marzano taxonomy, such content 
would be considered “information,” and part of the knowledge domain. That the information 
happens to be about the self-system would be of interest if the taxonomy were employed as part of 
a critical thinking curriculum, and organized by the mental process that operates upon it: here, 
simply comprehension; that is, student should comprehend - or perhaps simply recognize - that 
math as an academic activity requires certain personal dispositions and behaviors in order for the 
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student to find success. Again, in order to present findings organized by the topics of interest in 
this study - cognition, metacognition, and self-system - this attention to the knowledge domain is 
ignored, and what might be strictly be considered knowledge domain content about the self-system 
is instead organized as one of the processes of the self-system. 

How the Taxonomy of Educational Objectives Was Used to Classify Content 
Related to Cognition, Metacognition and the Self- system. 

Our analysts reviewed each standards document against each level within the taxonomy, providing 
excerpted text as evidence for the presence of related content appropriate to that level. Analysts 
focused on the knowledge, skills, and dispositions expected of students, whether this was 
communicated through explicit descriptions of content, through examples, student activities, or 
introductory sections intended for teacher guidance. The primary role of the content analysts, 
who have training in the analytic method and a degree or significant standards experience in the 
subject area they reviewed, was to make a fair inference as to the absence or presence of content 
within each document and to provide evidence for that judgment. Each assignment of content to a 
topic was reviewed by a second analyst. In the case of discrepant judgments, which were rare, the 
issue was resolved through deliberations or by a third analyst. (For the protocol employed, see 
Appendix C) 

The taxonomy was adapted to account for the variety of cognitive, metacognitive and self-system 
processes found in the standards documents through the development of topics that account for 
the variety of ways in which thinking skills (the cognitive system) appear across the documents. 

The elements added to the taxonomy for the purpose of represent content of the metacognitive 
and self-systems were developed from the categories common to the research and theoretical 
literature relating to these areas. For example, one aspect of metacognition is that it may be subject 
specific - as when students are reminded to monitor and self correct during the reading process - 
yet it might also appear generically stated, as when students are asked to refine and clarify their 
thinking. Such a distinction between subject-specific and generic descriptions of metacognition is 
commonplace in the literature. Similarly, metacognition may refer to the process of monitoring 
one’s cognition, or the knowledge one has about strategies for learning, or the characteristics of a 
task that affect cognition. Self-monitoring, strategy learning, and task analysis are also categories 
that appear in the literature (for a discussion, see Appendix A). In order to reflect the degree to 
which any one document holds expectations for students in these aspects of metacognition and 
the self-system, their content was organized against topics developed in this way. Such an approach 
provides a way to economically represent to what degree any one document touches on shared 
aspects of metacognition and the self-system, even when the documents otherwise have widely 
varying approaches in their treatment of these two systems. 
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APPENDIX A. A SURVEY OF COGNITION, METACOGNITION, AND 
SELF- SYSTEM IN THE LEARNING PROCESS 


The principal interest of this study is to identify the extent to which standards documents address 
student skills related to cognition and learning. In order to identify a taxonomy that would 
adequately address all salient aspects, it was necessary to identify the ways in which these terms and 
phrases are used and discussed. Differing conceptions about these topics became apparent during 
a review of the literature. 1 Three categories capture the range of discussion on student thinking 
and self-directed learning, namely: cognition, metacognition, and affect, or motivation (also known 
as the self-system). Two overviews of the research (Pintrich, 2000; Yuruk, Ozdemir, Beeth, 2003) 
touch on all three separate aspects. 

Cognition 

There are several taxonomies of educational objectives that can be or have been adapted to 
organize and specify the range of cognition, or thinking skills. Bloom’s taxonomy is the most 
widely known (named for the principal editor of this highly influential work) and is still in 
common use today. 

Bloom's Taxonomy 

The oldest and the most widely-known educational taxonomy, Bloom’s taxonomy, identifies a set 
of broadly defined cognitive skills. The taxonomy is problematic for two principal reasons. First, 
the levels of cognition are argued to be organized hierarchically by levels of relative difficulty. In 
Bloom’s design, levels (such as retrieval, comprehension, evaluation, and the like) are distinguished 
by degrees of complexity, on the assumption that complex tasks are inherently more difficult. 
However, it has been well established that very complex processes can be learned at a skill level 
that demands little or no conscious effort - that is, to a level of automaticity. Using such a 
taxonomy for standards analysis, then, would communicate that some documents or subject areas 
are more cognitively challenging than others, yet the research on Bloom’s structure does not 
support such a hierarchy. In addition, the identification of knowledge as equivalent to and 
including cognitive processes creates a category confusion that renders the taxonomy problematic 
for distinguishing content within the standards. Further, as a reflection of the level of work done 
on the topic in Bloom’s day, the taxonomy provides no means for categorizing metacognitive 
knowledge or processes, let alone processes related to the self in learning. 


1 We review studies published since 2000 to ensure that the overviews and theoretical work would address established 
research and theories that have established some authority or influence. We also included studies published before 
2000, when it appeared clear that they were seminal in the development of the constructs in which we are interested 
and added new information not available in the original set of documents. 
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Bloom’s taxonomy organizes content into six levels by the difficulty of the mental processes in 
which the student engages: 

1.00 Knowledge 

2.00 Comprehension 

3.00 Application 

4.00 Analysis 

5.00 Synthesis 

6.00 Evaluation 

Although organized as one among a list of cognitive processes, “knowledge,” of course, is not 
usually considered a process, but information on which or about which cognitive processes 
operate. Although Bloom defines knowledge as including “those behaviors and test situations 
which emphasize the remembering, either by recognition or recall, of ideas, materials or 
phenomena,” (Bloom et ah, 1956, p. 62), he includes within it the objects of the process as well, 
for example, the generalizations, principles, and the like that may be remembered or recalled (for a 
full discussion see Marzano & Kendall, 2007, p. 5 ff.). 

A recent taxonomy has as its subtitle “A Revision of Bloom’s Taxonomy of Educational 
Objectives” (Anderson et ah, 2001) and expands on the original. 

Anderson and Krathwohl, et al., Taxonomy 

A recent revision of Bloom’s taxonomy (Anderson et ah, 2001) acknowledges and addresses the 
category problem through the creation of a knowledge dimension as distinct from a cognitive 
process dimension. In addition, metacognition is added to the taxonomy. It is identified as 
knowledge and “awareness,” however, not as a process. An aspect commonly associated with 
metacognition— monitoring— appears in this taxonomy under the category of Evaluation: Checking, 
which effectively means that the activity of monitoring one’s own actions and learning is 
functionally equivalent to the monitoring of any event or behaviors not related to the self. This 
conflation weakens the taxonomy as a tool for identifying types of self-monitoring activities. 
Metacognitive knowledge is also defined to include references to the self as learner, which means 
that metacognition, in this scheme, encompasses but does not identify a number of potentially 
useful categories, such as motivation and student examinations of self-efficacy, which appear as 
important constructs in the literature on self-regulation. This limits its usefulness for identifying 
the full scope of material that appears in the research literature on metacognition and other skills 
associated with learning-to-learn. 

Cognitive processes are organized in the following categories: 

1. Remember 

2. Understand 

3. Apply 

4. Analyze 

5. Evaluate 

6. Create 
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Beneath each category are from 2 to 7 cognitive processes, for a total of 19, from recognizing in the 
first category to producing in the last. The taxonomy is also two-dimensional, the knowledge domain 
forming the second axis. This domain comprises factual, conceptual, procedural, and 
metacognitive knowledge. The distinction of knowledge and process addresses a problem in the 
original work and greatly increases the utility of this work. However, Anderson et al. identify 
metacognition as a type of knowledge, “knowledge of cognition in general as well as awareness of 
one’s own cognition” (p. 46). Awareness connotes self-monitoring - a process rather than 
knowledge - thus, this taxonomy does not include a means for identifying the processes of 
metacognition. 

The taxonomy adapted for this study (Marzano & Kendall, 2007), also uses a two-dimensional 
structure to improve upon Bloom’s original work. The structure is somewhat more complex than 
Anderson et al.’s design, but has a number of similarities. There are four levels of cognitive 
processing: 

1. Retrieval 

2. Comprehension 

3. Analysis 

4. Knowledge Utilization 

Beneath each level are from 2 to 5 cognitive processes for a total of 14, from recognizing in the first 
level to investigating in the last. (For the structure of the taxonomy, see table 1; for an overview, see 
figure 6.) In addition to the first four levels, which make up the cognitive system, the taxonomy 
adds another two: 

5. Metacognitive System 

6. Self-system 

These additional systems include four processes each, bringing to 22 the total processes available in 
the taxonomy. As we will note in the discussion on metacognition and self-system, the scope of 
categories in this taxonomy allows a level of analysis not available in Anderson or any other 
approach. In this taxonomy, metacognition represents a type of processing that is applied to 
subject matter content. In this approach, metacognitive knowledge is treated as the “subject 
matter” for mental processing, and therefore both metacognitive processing and metacognitive 
knowledge may be categorized using this taxonomy. 

Metacognition 

On the topic of metacognition the field is still somewhat inchoate and unsettled. There emerges 
no clear definition of metacognition that would reflect a consensus view among the studies. 
Metacognition can be described as knowledge or awareness of learning, as is the case for the 
Anderson &. Krathwohl taxonomy, or may include this knowledge as well as including a student’s 
regulation or control over tasks (see Baker & Cerro for a discussion). However, as a recent survey 
of models of self-regulation demonstrates, metacognition can also appear organized as one of a 
number of processes - including cognition, behavior, and affect - that the student regulates in the 
process of learning (Pintrich, 2000). Metacognition may also refer to monitoring or evaluating 
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one’s progress during an activity (Donovan, M.S., & Bransford, J. D., 2005, p. 10; Pintrich, 2000, 
p. 459; Prins, Veenman, and Elshout, 2006, p. 375). The term is also used to refer to activities or 
learned-strategies that are content-bound and acquired exclusively in a subject-specific context 
(Balin, 2002; Kuhn & Dean Jr., 2004, p. 270; National Research Council, 1999, p. 15). However, 
other characterizations of metacognitive strategies assert they may apply across all areas (White & 
Frederiksen, 2005), or more than one subject-area, though perhaps not every area (Bransford et ah, 
2006, handbook, p. 31). Researcher Davies (2006) succinctly captures the issue as a debate 
between the specifists— those who stress the importance of critical thinking understood as a subject- 
specific discourse— and the ‘generalists’— those who stress the importance of critical thinking 
understood independently of disciplinary context. For example, while listening to a political 
speech for a civics class, or the dialogue in a film, a student would profit by actively considering the 
speaker’s perspective and motivation, while such monitoring would be less useful for the student 
in a mathematics lecture. 

As noted in the discussion of taxonomies, the Anderson et al. taxonomy, although it alludes to the 
awareness component of metacognition, organizes metacognition as a type of knowledge, rather 
than a mental process. In this taxonomy, metacognition may be one of three types: strategic 
knowledge - for example, knowledge of the use of heuristics; task knowledge - for example, 
knowledge of the cognitive demands of different tasks, and self-knowledge, for example, knowledge 
that writing essays is a personal weakness (p. 46). This design recalls an influential model of 
cognition developed by Flavell (1979), who identified types of metacognitive knowledge in terms of 
strategic, task, and person variables. 

The Marzano & Kendall taxonomy organizes the metacognitive system by means of the following 
mental processes: 

• Specifying Goals: establish clear learning goals for specific types of knowledge 

• Process Monitoring: determine how effectively a plan or procedure is being 
carried in real time 

• Monitoring Clarity: determine the extent to which one is clear about specific 
aspects of knowledge 

• Monitoring Accuracy: determine the extent to which one is correct in terms of 
understanding specific knowledge 

In summary, metacognition may refer to the process of monitoring or of regulating one’s 
cognition, or the knowledge one has about strategies for learning, or characteristics of a task that 
affect cognition. Another aspect, the role of the self - specifically, motivation and affect - is 
addressed in the next section. 

The Self-system: Motivation and Affect 

Most models of metacognition include a component about the self in relation to what is being 
learned. The earliest example is Flavell’s influential model (1979), which organized metacognition 
by aspects of person, task, and strategy. As we found to be the case in the literature regarding 
metacognition, there are differing ways in which theoreticians and researchers think about what 
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Marzano & Kendall call die self-system: students’ attitudes, beliefs, and motivations. Discussions 
of motivation for learning and the self in relation to in the research literature often address 
student’s motivation and concept of self as learner (White, B. & Frederikson, J. 2005). 

The constructs most commonly associated with the idea of the self in relation to learning are self- 
directed and self-regulated learning. These terms are not clearly distinguished in the literature -of 
the two, self-regulation appears the preferred term, but conceptions of self-regulation are far from 
unitary - there appears no emerging consensus regarding what it may include (see, for example, 
Boekarts, 1995; Linnenrink, E. A., & Pintrich, P. R. (2004); Zimmerman B. J., & Schunk, D.H., 
2004). In an examination of the commonalities found among models of self-regulated-learning, 
Pintrich (2000) identifies one feature that we have earlier identified as associated primarily with 
metacognition - namely, the capacity to monitor aspects of cognition. Self-regulation also 
concerns those aspects over which students have actual or potential control, which includes 
motivation/ affect and behavior. In the Marzano & Kendall taxonomy these are addressed in this 
way: 


• Examining importance: analyzing the extent to which one believes that learning 
specific knowledge is important and then examining one’s beliefs relative to 
that issue. 

• Examining efficacy: examining the extent to which one believes one can 
improve one’s own understanding or competence relative to a specific type of 
knowledge. 

• Examining emotional response: identifying what emotions, if any, one 
associates with specific knowledge, whether these emotions interfere with 
learning, and the logic behind those associations. 

• Examining motivation: examining overall motivation to improve one’s 
understanding of or competence in a specific type of knowledge. 

Thus, the Marzano & Kendall taxonomy structures this exclusively as mental processes - self- 
system thinking. Yet not all characteristics associated with self-regulation in the literature or the 
standards documents reviewed can be characterized as a process. For example, the statement that 
students “should understand math is an academic activity that requires time, sustained 
engagement, patience, and persistence” (Conley, 2004, p. 31) could not be categorized as one of 
the processes outlined above, yet addresses the self-system. Adaptations in the Marzano & Kendall 
taxonomy were made in order to account for this, as described in the section Tool for analysis: A 
taxonomy of education objectives. 

In sum, the self-system includes those areas related to motivation and affect. These include aspects 
of interest or importance, personal efficacy, emotional response, and motivation. 
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APPENDIX B: DATA SOURCES AND THE LIMITATIONS OF THIS 
STUDY 


In order to identify the thinking and learning skills expected of all students, analysts reviewed 
three types of documents: state standards, national subject area standards, and standards that 
communicate the expectations held for graduating students by post-secondary institutions, 
including colleges and the high-skills workplace. For an overview of the source documents by 
subject area, see table Bl; for a bibliography of all the standards documents analyzed, see the end 
of this section. 

Document Sources 
S tates Standards 

The state standards selected for analysis were those of states in the Central Region. States 
commonly establish expectations for students in the form of academic standards. These standards 
describe, with some specificity, what students should know and be able to do by the time they exit 
12th grade. Such standards are commonly developed by and represent a consensus of stakeholders 
in the success of schools. These include not only educators, but parents and community and 
business leaders. In addition, state standards, either by policy, legislation, or both, have significant 
influences on daily instruction. Such standards are made readily available not only to teachers but 
to students, parents, and others through postings on the web sites of state education agencies. For 
this study, we identified the standards currently in force in the Central Region states for the four 
core subject areas— English language arts, mathematics, science, and the social studies. 

National Subject Area Standards 

Like state standards, national subject-area standards represent the consensus of numerous 
professional educators in the subject areas, typically at all levels of the education system. 
Representatives of business, industry, and the broader community typically contribute to the 
establishment of these standards. For this study, we selected significant and well-known documents 
representing what is commonly accepted as the national standards in the subject areas of language 
arts, mathematics, sciences, and the social studies. 

Expectations from Post-secondary Institutions 

Finally, in order to identify the expectations held in common by many employers of high-skilled 
workers and post-secondary institutions for high school students, we also examined studies 
produced by the American Diploma Project, The College Board, and Standards for Success. These 
documents identify the knowledge and skills that have been identified as essential for students to 
master if they are to succeed in post-secondary and the high-skills workplace. Like state standards, 
such documents describe what students need to know and do, in terms of specific knowledge and 
skills. Taken together, they represent the consensus of hundreds of professionals in universities 
and business and industry regarding what they believe students should have acquired by exit. 
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These studies are also national in scope, rather than specific to a region, a particular university, or 
an industry. 

The American Diploma Project (ADP) shares a similar purpose— connecting secondary and 
postsecondary expectations for success— but its focus is on “what it takes for graduates to compete 
successfully beyond high school— either in the classroom or in the workplace” (2004, p. 1). The 
Project, a partnership of Achieve, Inc., The Education Trust, and the Thomas B. Fordham 
Foundation, spent nearly two years working with two- and four-year postsecondary faculty and 
front-line managers in high-growth, high-skill occupations to define the core knowledge and skills 
that high school graduates need in order to be ready to succeed in their organizations. The results 
of the study are presented in Ready or Not: Creating a High School Diploma that Counts (ADP, 2004), 
which describes student expectations (termed “benchmarks”) for English and mathematics. The 
report also includes sample workplace tasks and post-secondary assignments, which illustrate in 
real terms how the knowledge and skills captured in the benchmarks might be applied beyond 
high school, whether in the workplace or in the college classroom. 

The College Board has developed a set of standards for English language arts and mathematics to 
assist schools in providing the “rigorous education that will prepare them for success in college, 
opportunity in the workplace, and effective participation in civic life” (p. iv). “The purpose of the 
project is to vertically align curriculum, instruction, assessment, and professional development 
across six levels beginning in middle school leading to AP and college readiness. The College 
Board Standards for College Success is, therefore, more specific than most standards documents 
because it is intended to provide sufficient guidance for curriculum.” (p. vi). 

Standards for Success, commonly abbreviated as S4S, is a project sponsored by the Association of 
American Universities in partnership with The Pew Charitable Trusts. One of the project’s 
primary goals was to identify what graduating high school students need to know and be able to do 
in order to succeed in entry-level university courses. These student expectations, termed 
“Knowledge and Skills for University Success,” are presented in Understanding University Success 
(Conley, 2003), the product of a two-year study in which more than 400 faculty and staff members 
from 20 research universities participated in extensive meetings and reviews. The disciplines 
covered included English, mathematics, natural sciences, social sciences, second languages, and the 
arts. (McREL participated as a consultant in this project.) 

Limitations 

The chief limitations in this study relate to the numbers and types of documents analyzed, which 
varied with each subject area; the level of analysis required to identify thinking and reasoning 
skills. Furthermore, some adaptations of the taxonomy were necessary in order to present the 
findings of our analysis. 

Each subject area differed markedly in the number and types of documents available for analysis. 
Because we selected the four core subject area standards for each of the seven states in the Central 
Region, the same number of state documents were subject to analysis for all subject areas. 

However, the number of national documents and documents that identified standards for post- 
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secondary success differed considerably. The subject areas of language arts and mathematics each 
have but a single document published by a nationally recognized body, while the social studies has 
five documents, and science, two. For post-secondary (college readiness), by contrast, the language 
arts and mathematics each have three documents, while science and the social studies have a single 
document. (See table B 1 for a summary; a list of documents analyzed follows that table.) 

Another limitation was the level of scrutiny that was required to identify the presence of thinking 
and learning skills in these documents. Content directly related to subject-matter was readily 
identifiable and organized under predictable headings— such as editing in the language arts, or 
biology in science. By contrast, content related to thinking and learning skills was often subsumed 
within discussions of the content itself, and so could only be identified after careful analysis. Even 
when the standards addressed thinking and learning skills more deliberately, it was rare that such 
discussions were signaled by text headings or other organizational cues. 

Finally, some adaptations had to be made to the taxonomy used to analyze the content. In the 
interests of identifying and coding all content related to metacognition and the self-system in a 
format that was meaningful across all documents, the method used to develop aspects or elements 
of each topic within these levels was different from that used for developing aspects for each 
cognitive process. Instead of developing a set of elements for each process directly from the 
standards documents themselves, as analysts did for the cognitive processes, we organized the 
elements of metacognition and the self-system primarily from the ways in which they are 
commonly addressed in the research literature— for example, to account for subject-specific 
metacognitive knowledge as well as metacognitive processes that are believed applicable to any 
subject area. (See Appendix A for a discussion.) We found that, perhaps because of an incomplete 
appreciation of the role of metacognition and self-system within standards, there were as many 
different aspects about these topics as the number of documents that addressed them. In order to 
keep the organization of findings easier to communicate, we elected to use categories common in 
the research rather than the many different categories that would arose from the documents 
themselves. 

Table Bl. Sources of documents analyzed for the study by subject area. 

Note: The number in brackets following the name of the authoring organization is the document 
identifier used in report figures. Full citations follow this table. 



English 
Language Arts 

Mathematics 

Science 

Social studies 

State standards 

All seven Central Region states [S1-S7] 

National Council 
of Teachers of 
English (1995) 
[Nl] 

National Council 
of Teachers of 
Mathematics 
(2000) [Nl] 

National Research 
Council (1996) 
[Nl] 

American 
Association for the 

Center for Civic 
Education (1994) 
[Nl] 

National Center 
for History in the 
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Advancement of 
Science(Project 
2061, 1993) [N2] 

Schools (1996) 
[N2] 

National Council 
on Economic 
Education (1997) 
[N3] 

National Council 
for the Social 
Studies (1994) 

[N4] 

National 
Geographic 
Research & 
Exploration (1994) 
[N5] 

Expectations 

American 

American 

Standards for 

Standards for 

related to 

Diploma Project 

Diploma Project 

Success (Conley, 

Success (Conley, 

college 

(2004) [Cl] 

(2004) [Cl] 

2003) [Cl] 

2003) [Cl] 

readiness/ 

The College 

The College 



high-skills 

Board (2006a) 

Board (2006b) 



workplace 

[C2] 

[C2] 




Standards for 

Standards for 




Success (Conley, 

Success (Conley, 




2003) [C3] 

2003) [C3] 
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APPENDIX C. PROTOCOL USED FOR THE ANALYSIS 


The protocol for analysis of the documents using the taxonomy consists of four major elements, as follows: 

1. Examine each document for any statement that indicates that students should engage at the cognitive, 
metacognitive, or self-system levels. Standards are the primary text for analysis, but introductory front 
matter (i.e., preface, introduction, rationale) should also be reviewed and included if this is where the 
content is solely or most clearly addressed. 

2. Excerpt the text of such statements and place it, along with the page number, in the input grid. This 
excerpt provides evidence for the presence of content and the means for quality assurance review by 
other analysts. 

a. Input Grid: Columns indicate subject area and source document (e.g., math: NCTM). Rows 
indicate the taxonomic level and sublevels. The taxonomic level is organized by system (such as 
cognitive system), the category within that system (such as knowledge utilization), the process 
within that category (such as Decision making). For the cognitive system, a subtopic is assigned 
or created by the analyst. For example, the following text is from North Dakota Science, grades 
11 - 12 : 

i. Explain how designing and implementing technology requires weighing trade-offs 
between positive and negative impacts on humans and the environment E64. 

b. The excerpted text from this example would be placed in the grid at the intersection of North 
Dakota science and Cognitive System: Knowledge Utilization: Decision Making: Consequences 
and Trade Offs. 

c. Topics in Cognitive System: In the cognitive system, create a topic based upon the focus of the 
benchmark. The Consequences and Trade-offs topic identified above was based on the aspect 
of decision making that was the focus of the benchmark. Before creating a new topic, review all 
previously created topics should be reviewed to determine whether they may address the 
material. If a topic should be revised so that it can better accommodate other material, analysts 
will negotiate topic revisions. 

d. Categories in the Metacognitive and Self-System: We find that the metacognitive and self 
systems are sporadically addressed in the documents and, as a consequence, the topics they 
address are too diverse to be useful in organizing the material. In other words, each excerpt 
typically presents a unique topic for the process, so there are as many topics as there are 
instances. Instead of organizing processes by topics as in the cognitive system, the processes are 
organized by types. For the metacognitive and self-system, assign content to one of the 
following three types: 

i. Content is applicable across subject areas (for example, “learners understand the 
importance of reflecting on their thinking”) 

ii. Content is made specific to this subject (for example, “it is important to understand 
the connection of mathematics to life outside the classroom”) 
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iii. Content relates primarily to student disposition (for example, “the student develops 
flexibility and perseverance”). 

e. Evidence for students’ engagement at a topic (or type) within a taxonomic level only need be 
established once for each document. For example, it is trivial to establish that students are 
expected to retrieve information about science. There will likely be hundreds, if not thousands, 
of examples that indicate that students are expected to recall information. This only need be 
coded once for each document. For example, evidence of recall in a science document would 
be categorized only once for that document under Cognitive System: Retrieval: Recalling 

3. Format 

a. Edit large statements to delete content that isn’t relevant to the question. 

b. Split up statements that include ideas related to several topics/types and using ellipsis to 
indicate content has been moved elsewhere or deleted. If deleting content would create a vague 
or confusing statement, use bold text to indicate what content is of interest for the current cell 
assignment. 

4. Quality Assurance 

a. To conduct a review of another analyst’s work, first review all mappings and topic/type 
assignments. If you disagree with a mapping or topic/type assignment, confer with the original 
analyst. If you cannot resolve the issue, make a note in the cell. The Principal Investigator (P.I.) 
will assign a third analyst or conduct the additional review, making edits with rationale. We 
have been able to reach consensus on all coding. 

b. For those cells that are empty, review the original standards document to satisfy yourself that 
indeed the content does not appear. If you find material that you believe is evidence that the 
skill is covered, provide the excerpted text in the appropriate skill level and topic/type. This 
will be reviewed by the original analyst & the P.I. 
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